Component-based stream processing "in the cloud"

In earlier work, we reported on modeling of stream processing in terms of distributed components (as exemplified in the EU CoreGrid project ProActive), showing how a stream processing system can be built from components in composition, with dynamic reconfiguration and distributed management of the streams. In this paper, we introduce the Web Service Stream Deployer (WSSD), for the remote establishment and deployment of streams across widely distributed resources, allowing a user to set-up, control and reconfigure a stream remotely and dynamically. We demonstrate this concept with streams on a cloud testbed, using Nimbus cloud infrastructure at the University of Chicago. In particular, we show that our web-services based WSSD exhibits minimal adverse latency effect when used over an intercontinental network to manipulate, from our client in Australia, a cloud-based stream in the USA; we believe that this represents a useful mode of remote interaction with cloud-based applications. We outline some ideas in scheduling of cloud-based streams; we present a model of interaction with Gridbus, a widely used framework for exploring different scheduling algorithms (such as economic scheduling), whereby scheduling criteria expressed in Gridbus can be used to automatically deploy stream components.Paul N. Martinaitis, Craig J. Patten, Andrew L. Wendelbor